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METHOD AND APPARATUS FOR 
GENERATING A DISPLAY SIGNAL. 
This invention relates to a method and apparatus for 
interactively generating a display signal. 
5 Traditional TV broadcast is characterised by temporal 

synchronisation for all viewers, ignoring the small 
differences in transit times of the signal due to 
differences in transmitter to user distances. Therefore at 
the moment of broadcast all viewers receive the same, 
10 uniform signal substantially instantaneously. Interactive 
forms of the medium, such as Two Way TV, Web TV are 
generally characterised by providing the viewer with the 
facility to interact with a designated interactive area of 
the TV screen. This may utilise a split screen in which 
15 the interactive area may have internet information, an on 
screen menu, or information which may be relevant to the 
main broadcast part of the TV screen. These systems are 
characterised by presenting information in response to the 
user's interaction within a fixed predefined interactive 

2 0 area of the screen. 

In contrast to the limited interaction with TV 
broadcast on current systems, computer games consoles (eg. 
Sony Playstation, Sega Saturn, Nintendo 64) are presented 
to the viewer on TV screens or dedicated display screens, 

25 but each viewer has the ability to achieve unique 
interaction with the imagery/ sounds presented on the TV, 
which are processed by the games console using computer 
technology, in response to the users actions. Generally 
the user's perceived TV environment (including images, 

30 sounds and other sensory signals) are produced by computer 
generation within the games console, which may or may not 
also utilise digitised predefined data, such as sounds in 
the TV environment which shall be referred to as the 
computer generated environment (CGE) or in the specific 

3 5 case of images, computer generated imagery (CGI) . These 

games consoles and the CGE are characterised by high 
frequency update rate, typically in excess of 25 frames per 
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second or 25H2. This gives the impression of instantaneous 
or real time response to the user's actions and also a 
smooth and seamless dynamic image. The individual frames 
are not discernible as individual frames, but rather 
5 contribute to the overall real time dynamic environment, 
giving the impression of real time control and interaction 
with the CGE. 

It is desirable that the real time CGE is as realistic 
as possible, and greater degrees of realism are generally 
10 achieved by increased computer processing power and by 
using the most efficient representation in terms of realism 
versus processing power. By way of example only, one of 
the most efficient representations for CGI uses relatively 
coarse polygonal or faceted geometric model in which the 
15 greatest detail in terms of polygon distribution would 
generally be used in the more geometrically complex areas. 
By a process known as texture mapping, in which 
photorealistic textures representing surface features are 
mapped onto the individual polygon faces, a relatively 
20 realistic CGI is achieved notwithstanding the relatively 
coarse polygonal geometry representation. The product of 
the number of texture mapped polygons in the CGI and the 
image update rate measured in frames per second yields a 
number defining the number of texture mapped polygons the 
25 computer processing is required to process per second, 
which may be 1,000,000 polygons per second. 

We have identified that the level of interaction 
offered to the viewers of interactive broadcast TV is 
limitred, and this consequently limits the applications of 
3 0 such medium. Furthermore we have identified that although 
the interaction of a games console is greater than 
interactive TV, the actual theme of the CGE is limited to 
the specific game content loaded into the console, for 
example via CD ROM, cartridge, the internet or broadcast to 
3 5 a TV with the appropriate hardware to run CGE type games or 
by other means . 
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In accordance with a first aspect of the present 
invention there is provided apparatus for interactively 
generating a display signal, the apparatus comprising 

a receiver for receiving a broadcast signal , the 
5 broadcast signal comprising a plurality of datastreams each 
including image data defining a background object, and 
control parameters ; 

a foreground computer generated imagery (CGI) device 
for generating a foreground computer generated object 
10 (CGO) ; 

a mixer for combining the foreground CGO with 
background object image data from the receiver to generate 
the display signal; and 

interaction means for receiving the control parameters 
15 from the receiver, monitoring the position of the 
foreground CGO, and adapting the display signal with 
reference to the monitored position of the foreground CGO 
and the received control parameters. 

In accordance with a second aspect of the present 
2 0 invention there is provided a method of interactively 
generating a display signal, the method comprising 

receiving a broadcast signal, the broadcast signal 
comprising a plurality of datastreams each including image 
data defining a background object, and control parameters; 
2 5 generating a foreground computer generated object 

(CGO) ; 

combining the foreground CGO with the background 
object image data to generate the display signal; 

monitoring the position of the foreground CGO; and 
30 adapting the display signal with reference to the 

monitored position of the foreground CGO and the received 
control parameters . 

The present invention provides the capability of 
interaction with the actual broadcast itself as it appears 
35 on a screen in real time. 

Typically the method and apparatus is provided for use 
in association with a TV set to provide levels of 
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interaction with broadcast content that is normally 
associated with a games console, whereby such interaction 
is achieved within the main broadcast as presented on their 
TV screen. This provides the user or viewer with a CGE, 
5 hereinafter referred to as the foreground computer 
generated object {foreground CGO) which interacts with the 
broadcast signal which by way of example only may be 
delivered by Digital Terrestrial, Satellite or Cable 
broadcast medium and in which the background object 
10 typically changes continuously during the broadcast 
transmission, not only from programme to programme but from 
frame to frame at a rate of, by way of example only, in 
excess of 20Hz. 

Advantageously this invention offers full screen 
15 interaction via the foreground CGO with objects represented 
in the broadcast, for example but not exclusively visible 
objects, and such interaction is assured even though 
objects may change position, shape, motion, behaviour at 
the broadcast update rate, and furthermore the nature of 
20 the interaction with such objects may also change at the 
broadcast rate . 

The term "broadcast" as used herein is intended to 
cover all transmission media, including but not 
exclusively, digital terrestrial, cable, satellite 
25 broadcast to all display devices, including but not 
exclusively, TV, computer, or games console. The broadcast 
is typically a mass broadcast {ie. the signal is broadcast 
simultaneously to a plurality of TVs, computers or game 
consoles) , The signal may also be broadcast via the 
30 Internet. The term "datastream" as used herein refers to 
different broadcast content relating to the same programme, 
and all datastreams are broadcast substantially 
simultaneously. The term "programme" refers to a set of 
such multiple datastreams which relate to the same 
35 programme, where each datastream within the set of 
datastreams making up the programme broadcast are 
temporally synchronised, and relate to the same content 
but, by way of example only, offer an alternative view of 
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the content. The term "CGE" as used herein is intended to 
cover a multimedia representation, including but not 
exclusively still images, dynamic images, sounds, real time 
images and real time audio signals. The term "foreground 
5 CGO" , "foreground CGE" or "foreground" as used herein is 
intended to relate to any and all representations which are 
not part of the broadcast, but are computer generated and 
which may be displayed on the TV screen, by way of example 
but not exclusively overlaid on the broadcast or in a 
10 separate interactive area of the screen. Alternatively the 
"foreground CGO" "foreground CGE" or "foreground" may apply 
to such representations which are not part of the broadcast 
and are not displayed or otherwise represented on the TV 
screen. By way of example only, portions or sections of 
15 the foreground CGO may be hidden from the user or viewer 
for the purpose of acting as geometric reference to 
calculate interaction between the foreground CGO and the 
background object. The term "computer" in the context of 
"computer generated", "computer processing", "computer 
20 generated imagery" or "computer generated environment" 
refers to any apparatus, equipment, hardware, software, 
parts thereof and combinations thereof which processes the 
foreground CGE, and by way of example only may be a 
computer, a set top box (as produced by General 
25 Instruments, Pace Micro Technology by way of example only) , 
a games console (as produced by Sony, Sega, Nintendo by way 
of example only), parts or sections thereof, or customised 
hardware including but not exclusively computer memory, a 
processor and an optional graphics processor. The term "TV 
30 screen" or "display screen" as used herein is intended to 
cover any display device or system or assembly in which 
there is a display element including but not exclusively, 
TV screen, computer monitor, projection system, head 
mounted display. The term "receiving hardware" as used 
3 5 herein refers to any apparatus, equipment, hardware, 
software, parts thereof and combinations thereof which 
receives the broadcast datastreams, the receiving hardware 
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input from the broadcast medium and transmits the signals, 
the receiving hardware output to the mixer, and may by way 
of example only be referred to as a decoder, and by way of 
example only may be a computer, a set top box (as produced 
5 by General Instruments, Pace Micro Technology by way of 
example only) , a games console (as produced by Sony, Sega, 
Nintendo by way of example only) , parts or sections 
thereof, or a customised hardware including computer 
memory, a processor and an optional graphics processor. 

10 The term "nominal user position" or "user position" defines 
a distance measured normal from the plane of the TV screen 
in a normal direction, to the position of the user, where 
this distance and the user position are used purely for 
calculation purposes, and impose no further restriction on 

15 the actual user position in addition to the everyday 
physical constraints. The terms "viewer", "viewers", 
"user" and "users" can be exchanged and interchanged with 
no loss of generality. The term "controller" or "hand 
controller" as used herein refers to any device with which 

2 0 the foreground CGO and interaction with the background CGO 
is controlled by the user, including but not exclusively 
motion and interaction, requests to upload and download 
other data or information. By way of example only, the 
controller may be an infrared device operated by buttons or 

25 direct voice activation. 

Advantageously, with the broadcast signal, there are 
multiple datastreams relating to the programme, each 
datastream representing, by way of example only, an 
alternative view, a user selectable view, additional 

30 relevant information pertinent to the corresponding frame 
on one or more of the other datastreams, where each 
datastream is updated at, by way of example only, 25Hz. 
The multiple datastreams may be time-division-multiplexed, 
ie. transmitted one after the other on a single frequency. 

35 Alternatively the datastreams may frequency-division- 
multiplexed, ie . transmitted simultaneously on different 
frequencies . 
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Advantageously, within the broadcast signal, control 
parameters are transmitted on each datastream, and 
temporally synchronised with each frame of the datastream 
broadcast, which may by way of example only be transmitted 
5 at a rate of 2 5 frames per second, and such control 
parameters define the interaction the user can achieve 
between the foreground CGO and the background CGO or the 
broadcast for each frame, and those control parameters and 
the associated defined interaction may vary at a rate of eg 
10 25Hz. The control parameters are used by the interaction 
means to define the interaction for that particular frame 
of the datastream broadcast . 

Typically the broadcast signal contains multiple 
datastreams relating to the same programme, and temporally 
15 synchronised control parameters. The broadcast is received 
by the receiving hardware, which by way of example only may 
be a set top box or part thereof . The receiving hardware 
transmits the decoded broadcast images to the mixer at a 
predetermined rate (eg 25Hz) . 

2 0 Advantageously, the control parameters are 

concurrently or simultaneously received by the interaction 
means, and for each frame of the broadcast signal the areas 
of interaction and the nature of such interaction is stored 
for each frame. By way of example, the areas of 
25 interaction are defined by the co-ordinates in three 
orthogonal axes resolved or projected into the 
substantially 2 dimensional plane of the TV screen with a 
predefined viewing angle, resulting in a 2 dimensional 
representation of the interaction areas overlaid on each 

3 0 frame of the broadcast and such frames and the associated 

interactive areas may change at the update rate, eg 2 5Hz. 
For example the viewing angle may be 90° in a horizontal 
plane, which in turn defines a nominal user position for 
computer calculations and projection from the three 
3 5 dimensional representation to the two dimensional screen 
representation. The viewing angle and the corresponding 
nominal user position define the extent of the broadcast 
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8 

background visible on the screen, and are preferably 
selected to complement and match broadcast standards for 
visible viewing angle. 

Advantageously the nature of the interaction for each 
5 area may include, but not exclusively, areas within the 
background that the user controlled foreground CGO can 
interact with, areas which the user can click with a cursor 
device, areas which are linked to information sources 
activated by clicking, the type of interaction between the 

10 foreground CGO and the interactive area. 

Advantageously, the processing within the foreground 
CGI device generates the foreground CGO, which is the user 
controlled representation by which the user interacts with 
the interactive areas defined by the control parameters for 

15 each frame. The foreground CGI device transmits the 
foreground CGO to the mixer to be overlaid on the 
background object within the broadcast signal for each 
frame of the broadcast (eg at a rate of 25 Hz) , whereby the 
position, shape and other features of the foreground CGO 

20 including but not exclusively colour, sound, direction of 
motion, visibility, as modified by the user interaction, 
are updated at the aforementioned rate. 

Preferably the mixer combines the representations of 
the background broadcast and the foreground CGO such that 

25 the foreground CGO is overlaid on the background broadcast 
and the background CGO. 

Advantageously there is processing available to 
determine which individual pixels within the foreground 
CGO, 'the background broadcast and any other displayed 

30 feature are closer to the plane of the TV screen when 
measured in the aforementioned three orthogonal axes 
defining a three dimensional geometric space. Furthermore, 
for each pixel position of the screen measured in the two 
dimensional screen co-ordinates, the pixel properties 

3 5 including but not exclusively colour, are those 
representing the properties of the object closest to the 
screen measured in the aforementioned 3 orthogonal axes at 
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the point projected onto the two dimensions of the TV 
screen. Advantageously geometric information relating to 
the background is conveyed in the control parameters 
defining the background CGO . Advantageously, parts or 
5 sections of objects including but not exclusively the 
foreground CGO, the background CGO and the broadcast 
background which when resolved into the screen co-ordinate 
system lie outside the screen dimensions or lie between the 
screen and the nominal user position, or are within the 

10 screen dimensions but further away from the screen than 
some other object are not displayed. This process is 
conventionally known as culling. 

Advantageously for material recorded in any format for 
subsequent broadcast, including but not exclusively Betacam 

15 or digitally stored images, the control parameters are 
encoded in or with the material prior to broadcast. The 
control parameters for each frame of the broadcast may 
include the areas of interaction defined in the two 
dimensional screen co-ordinates, the nature of the 

20 allowable interaction, the resulting action arising from 
such interaction. By way of example only, interaction with 
one of the areas may cause information to be displayed, or 
an alternative datastream of the broadcast to be displayed, 
or further information be displayed on the screen. 

2 5 Advantageously for live broadcast material or real 

time feeds, the control parameters are generated 
automatically or semi-automatically , eg utilising vision 
systems which interpret each frame of the broadcast as it 
occurs in order to identify particular features- within the 

3 0 frame image, and within such group of features areas that 

are to be automatically converted to interactive areas. 
The conversion from vision system identified features to 
interactive areas may be augmented by the use of technology 
including but not limited to, artificial intelligence, 
3 5 neural networks, knowledge based systems or combinations 
thereof, to generate such interactive areas and other 
control parameters subject to predefined rules based on the 
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10 

identification of the nature of the feature. Preferably, 
the rate at which the control parameters are generated is 
commensurate with live broadcasting, and by way of example 
only, this will be at a rate equivalent to the broadcast 
5 frame update rate (eg 25Hz) . For the purpose of 
illustration only, this technique used with live broadcast 
may be used for computer games based on live footage or 
training simulators based on live geographic features and 
scenarios . 

10 Advantageously, the type of foreground CGO may be 

selected by the user from a library of such foreground CGO 
stored in memory, or on a data storage device connected to 
the foreground CGI device which may be a DC ROM, or a 
Digital Versatile Disc (DVD) . Preferably the foreground 

15 CGO may also be downloaded t'o the foreground CGI device at 
the beginning or during the broadcast of the control 
parameters. Such definition of the foreground CGO will 
define features including but not exclusively, colour, 
shape, texture, allowable movement, sound effects, 

20 articulation on screen. 

Advantageously, the foreground CGO object movement and 
interaction is controlled via a user input using a hand 
controller (eg a standard or modified infrared controller) 
and subject to the rules defined by the control parameters 

25 for each frame. Alternatively other input devices may be 
used to control the foreground CGO and interaction 
including but not exclusively, voice activation, mouse, 
game controller or pad. 

•Advantageously, the nature of the foreground CGO will 

30 match the nature of the broadcast. By way of example only, 
a broadcast comprising a road or driving theme where the 
road representation itself is an interactive area may suit 
a foreground CGO based on a vehicle, such as a car. The 
car based foreground CGO will have motion dynamics 

35 representative of a real car, to the extent that a games 
console car is representative. By way of a further 
example, a broadcast comprising an exploration or tour 
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theme may suit a foreground CGO based on an articulated 
walking human figure, and the control parameters would 
define valid interactions, eg the ability to walk the 
foreground CGO through doors in the broadcast background, 
5 but not through walls. 

Alternatively in some broadcast programmes or themes, 
the foreground CGO may not match the broadcast, and may be 
an abstract representation of the position of the 
foreground CGO, such as a cursor, an arrow, or an icon 
10 graphic of a hand. 

When the user controls the foreground CGO with the 
controller, the foreground CGO may perform functions 
including translat ional motion and rotational motion about 
the three orthogonal axes and combinations thereof resolved 
15 into the two dimensional Screen co-ordinates, initiate 
sounds, or interact with defined interactive areas where 
such interaction results in further action (eg information 
presentation in text, graphic, video or mult i -media forms 
or combinations thereof) . 
2 0 Advantageously, the user has the option of downloading 

information from the broadcast, relevant to the broadcast, 
eg foreground CGO representations, additional information, 
software, control parameters. 

Preferably the user will also have the option of 

2 5 uploading information relevant to the broadcast programme 

by specific interaction between the foreground CGO and the 
background CGO. Such uploading is typically achieved by 
communication between the controller and the interaction 
meaner. By way of further examples, certain types of 

3 0 uploaded request may be considered as an uploaded control 

parameter, which may cause a switch to a different 
datastream of the broadcast containing additional 
information, whereas other uploaded requests may be for 
information not available within the broadcast, and such 
3 5 requests are routed to a World Wide Web site for the 
specific programme via telephone connection and modem. 
Preferably the user will also have the option of uploading 
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data relevant to the programme, eg performance scores 
achieved by the user in an interactive game scenario. 

By way of further explanation, it may be informative 
to consider the control parameters and the interactive 
5 areas defined by such parameters as a background CGO with 
which the user via the controller can cause the foreground 
CGO to interact with.- Preferably, but not essentially, the 
background CGO includes a coarse geometric representation 
and visual quality is greatly improved by the 
10 aforementioned technique of texture mapping whereby the 
broadcast image for each frame is a substantially full 
screen texture which we will refer to a Supertexture . 
Preferably, the geometry of the background CGO is not 
visible to the user. By way of further clarification the 
15 combination of the foreground CGO, the interaction defined 
in the control parameters, the background CGO defined by 
the control parameters and the broadcast Supertexture 
provide an interactive CGE based on the broadcast 
technology which is comparable with that achieved with a 
2 0 games console. 

The aforementioned combination of features provides 
the user with the ability to interact with the features 
within the broadcast. 

Advantageously, this invention may be used as the 

2 5 basis or foundation of a commercial service in which the 

user pays for usage, eg on a per programme or per unit time 
basis. Advantageously, such payment method may be 
incorporated into the apparatus, such as smart card 
operation, or an additional feature of the interaction via 
30 the controller, such as a user capability to enter credit 
card information which is then treated as uploaded 
information as previously described, using secure 
transaction protocols and techniques. 

An embodiment of the present invention will now be 

3 5 described with reference to the accompanying drawings, in 

which : 
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Figure 1 is a schematic diagram of an interactive 

systems- 
Figure 2 is a schematic diagram showing the 

compilation of the broadcast signal; and 
5 Figures 3-8 show different images displayed on the 

screen . 

Figure 1 shows a schematic of the apparatus and 
connectivity. The programme broadcast has an origin 1 and 
broadcasts multiple datastreams (a) - (d) in a mass broadcast 

10 signal 15 via a broadcast medium 7, which are received by 
receiver hardware 2 in a set top box 14. Only a single set 
top box 14 is shown but it will be appreciated that the 
mass broadcast signal 15 will be picked up by a large 
number of set top boxes at any one time. 

15 Figure 2 is a schematic drawing illustrating how the 

origin 1 compiles the broadcast signal 15. In Figure 2 
only two datastreams 20,21 (ie. datastream (a) and 
datastream (b) ) are shown but in a practical system a large 
number of datastreams will be compiled and broadcast 

2 0 simultaneously, with the number of datastreams being 

limited only by the bandwidth of the broadcast medium 7. 

Each datastream 20,21 comprises a series of frames of 
background object image data, and control parameters which 
control interactions with the background image data. The 

25 background object image data comprises a series of frames 
of live full screen broadcast video data (frames 22-27) and 
CGI image data 70-75 (referred to hereafter as background 
CGO) . In this example frames 22-24 are three subsequent 
live 'TV frames of video data showing a room viewed from a 

30 central position, and frames 22-24 are views of the same 
room viewed from a position at one side of the room. The 
background CGO 70-75 contains a computer generated 
description of the back wall 76,77 of the room, along with 
its 3D positional coordinates. The background CGO 70-75 

3 5 may be simply in the form of bitmap data or it may be a CGI 

programme which can be loaded into the set top box to 
generate a CGI image . 
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In addition each datastream comprises a set of control 
parameters including hot-spot coordinates 28-33 and hot- 
spot interaction rules 34-39. The hot-spot coordinates 28- 
33 comprise three-dimensional position coordinates which 
5 define the positions of "hot-spots" 40-45 in the room as 
viewed from the respective viewing positions of datastream 
(a) and datastream (b) . The hot-spot interaction rules 34- 
39 define the nature of the interaction between the 
foreground object and the hot-spots, as discussed below. 

10 The origin 1 also generates software motion models 50- 

52 (which define the relationship between the user inputs 
and the movement of the foreground object, as discussed in 
further detail below) and foreground supertexture data 53- 
55. In addition the origin 1 generates frame identifiers 

15 4 8 which are transmitted With each respective frame of 
information. For instance items 22,28,34,25,31 and 37 are 
all associated with the same frame and hence are given the 
same frame identifier . 

The datastreams are compiled by a compiler 4 9 to form 

20 the broadcast signal 15 as illustrated in Table 1. Table 
1 illustrates an example in which ten datastreams are 
carried by the broadcast signal 15. The broadcast signal 
15 comprises a series of time-division-multiplexed data 
packets which are transmitted in the order shown in Table 

25 1. The first two packets contain the supertexture data 53- 
55 and motion models 50-52. Packet 3 is a datastream 
identifier associated with datastream (a) . Packet 4 is a 
frame identifier associated with the first frame. Packets 
5-8 contain the data associated with the first frame of 

30 datastream (a) (ie. items 22,28, 34 and 70 from Figure 2). 
Packets 9-14 carry datastream (b) information for the first 
frame. Packets 15-62 (not shown) carry data associated 
with the eight other datastreams for the first frame 
(including packet 57 which is a datastream (j) identifier, 

35 and packet 58 which is a frame identifier associated with 
the first frame) . 
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The next frame of information is then transmitted, 
starting with packet 63 (datastream (a) identifier) and 
packet 64 (frame 2 identifier) . 

Although the background object data and hot-spot data 
5 is shown in Table 1 being transmitted at the same rate, it 
will be appreciated that in other cases (e.g. with a 
stationary background) the data may be transmitted at 
different rates. 

The non-video data 10 in the broadcast signal 15 (ie. 
10 the background CGO data 70-75, the hot-spot coordinates 3 0 
and the hot-spot interaction rules 34-39) are passed to a 
computer 3, and an initial datastream is selected by an 
upload request signal 11 from the computer 3 to the 
receiver hardware 2. The background object video data 8 
15 (eg. frames 22-24) in the selected datastream is decoded 
and transmitted to a mixer 4 . The computer 3 generates a 
foreground CGI image and a background CGI image (as defined 
by the programmes contained in the background CGO data 70- 
72) and the calculated CGI representation 16 is transmitted 
2 0 to the mixer 4. The foreground CGO, the background CGO, 
and the video data 8 are combined for every pixel on the TV 
screen 5 and the resulting combined signal 17 is 
transmitted to the TV screen 5. The multiple datastreams 
(a) - (d) , the control parameters 10, the selected datastream 
25 8, the upload request signal 11, the foreground and 
background CGO 16 and the combined signal are updated at a 
rate of, by way of example only, 2 5 times per second. The 
foreground CGO and interaction with the background CGO is 
controlled by the controller 6. Control signals from the 
30 controller are transmitted 9 to the computer, which by way 
of example only, may modify the foreground CGO 
representation whilst the background CGO is updated based 
on the updated broadcast control parameters 10 and the 
recalculated representation 16 is transmitted to the mixer 
35 17. Uploaded information or requests for information not 
contained within the broadcast will be routed 12 to a World 
Wide Web site 13 for the specific programme via telephone 
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connection and modem 18. In the case where the invention 
is used as the basis of a commercial service, the receiver 
hardware 2 is activated only after valid user payment has 
been made via the smart card device 21, and the control 
5 activation signal 22 has been sent to the receiver hardware 
2, thereby initiating the whole process. 

Figure 4-7 are views of the TV screen 5 during an 
interactive game. At the start of the game, the origin 1 
transmits the motion models 50-52 and supertexture data 53- 
10 55. The data 50-55 is passed to the computer 3 and stored. 
The user then selects a preferred character to play the 
game using controller 6. In this example the user selects 
a human character 56 associated with supertexture data 53 

and motion model 50. 

15 The datastreams are then transmitted as illustrated in 

Table 1. An example of a suitable data protocol is digital 
MPEG2 . Initially the computer 11 automatically generates a 
default upload request signal 11 (ie. without any input 
from the controller 6) which causes the receiver 2 to 

20 select and decode the default datastream (a) . As a result, 
the image on the screen 5 is as shown in Figure 3, ie. a 
background image 57 of a room as defined by the background 
object video data 22-24, with a back wall 78 generated from 
background CGO data 70-72, a hot-spot 41 (a door at the 

25 back of the room), a hot-spot 40 (a feature, eg a chair or 
table on the floor of the room) and a hot-spot 42 (a trap- 
door) . The computer 3 generates a coarse CGI 
representation of character 56, with further definition 
being provided by the supertexture data 53. Additional 

3 0 supertexture or motion models may be provided by a CD ROM 
or other storage device 19. The CGI character 56 is 
transmitted to the mixer 4, and overlaid on the background 
view 57. By operating controller 6, the user can move the 
character 56 around the room as indicated by the arrows in 

35 Figure 3, with the range of movement being defined by the 
motion model 50 associated with the character 56. 
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When the user presses a "view change" button on 
controller 6, an upload request signal 11 causes the 
receiver 2 to switch the selected datastream 8 to 
datastream (b) . The view on screen 5 then changes to the 
5 view 58 shown in Figure 4, as defined by the background 
video data 25-27 and background CGO 73-75. 

An alternative method of switching datastreams is 
shown in Figures 5 and 6. When the character 56 reaches 
door 41, the computer 3 senses that the character 56 is 
10 coinciding with a hot-spot 41 (with reference to the hot- 
spot coordinates 28-30) and downloads the hot-spot 
interaction rule (34-36) associated with hot-spot 41. The 
downloaded rule states "if character reaches door 41, then 
cause background CJO image to open door, and then switch to 
15 datastream (j)". Therefore • the computer 3 first generates 
a CGI image of the door 41 opening (under the control of 
the program downloaded from the broadcast GGO data (70-72)) 
and then generates an upload request signal 11 which causes 
the receiver 2 to switch to datastream (j) which contains 
20 a view 59 of a different room with hot-spots 60-62 (Figure 
6) . 

The process of "culling" is illustrated in Figure 7. 
The computer 3 knows the three-dimensional position of the 
character 56 in the room 57, and also knows the position of 

25 the hot-spot 40 (as contained in the hot-spot coordinates 
28-30) . If the character 56 is "behind" the hot-spot 40 
(as viewed from the current viewing position) then the 
computer 3 recalculates the representation 16 of the 
character 56 and does not transmit the obscured part of the 

30 character 56 to the mixer 4. 

Figure 8 illustrates an alternative, two-dimensional 
game. A background scene 80 is formed by a single bitmap 
of background CGO data. A pair of foreground CGI 
characters 81,82 in a fighting game are superimposed on the 

3 5 background scene 80, and can be moved around the scene 80 
by respective controllers. Three hot-spots 83-85 each 
display advertising material, and each has an associated 
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hot-spot interaction rule "characters should not pass in 
front of hot-spot and obscure advertising material". This 
prevents the computer 3 from causing the characters 81,82 
to pass in front of the hot-spots 83-85. In the case of 
Figure 8. the background is static and so the background 
video data and/or background CGO is only transmitted once 
during a game, instead of once every frame (as with a 
moving background) . 
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Packet No 



APPENDIX 
TABLE 1 



Packet Description 



character supertextures 
character motion models 
datastream identifier 
frame 1 identifier 
background object video data 
background object CGO data 
hot-spot area coordinates 
hot-spot interaction rules 
datastream (b) identifier 
frame 1 identifier 
background object video data 
background object CGO data 
hot-spot area coordinates 
hot-spot interaction rules 



datastream (j) identifier 
frame 1 identifier 



datastream A identifier 
frame 2 identifier 
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CIAIMS 

1. Apparatus for interactively generating a display 
signal, the apparatus comprising 

a receiver for receiving a broadcast signal, the 
5 broadcast signal comprising a plurality of datastreams each 
including image data defining a background object, and 
control parameters ; 

a foreground computer generated imagery (CGI) device 
for generating a foreground computer generated object 
10 (CGO) ; 

a mixer for combining the foreground CGO with 
background object image data from the receiver to generate 
the display signal; and 

interaction means for receiving the control parameters 
15 from the receiver, monitordng the position of the 
foreground CGO, and adapting the display signal with 
reference to the monitored position of the foreground CGO 
and the received control parameters. 

2 . Apparatus according to claim 1 wherein the control 

20 parameters define the position (s) of one or more areas of 
interaction in the background object, and wherein the 
interaction means adapts the display signal when the 
position of the foreground CGO coincides with the position 
of a selected area of interaction. 

25 3 . Apparatus according to claim 2 wherein the control 
parameters define one or more rules associated with the or 
each area of interaction, and wherein the interaction means 
adapts the display signal in accordance with the or each 
rule associated with the selected area of interaction. 

3 0 4 . Apparatus according to any of the preceding claims 
wherein the interaction means adapts the display signal by 
adapting the foreground CGO input to the mixer. 
5. Apparatus according to any of the preceding claims 
wherein the broadcast signal comprises a plurality of 

3 5 datastreams, the receiver transmits background object image 
data to the mixer from a selected one of the datastreams, 
the selected one of the datastreams being selected in 
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response to an upload request signal, and wherein the 
apparatus further comprises means for inputting upload 
request signals to the receiver in response to input from 
a user. 

5 6. Apparatus according to claim 5 wherein the interaction 
means adapts the display signal by inputting an upload 
request signal to the receiver. 

7 . Apparatus according to any of the preceding claims 
further comprising a user operable controller for 

10 controlling the foreground CGO generated by the foreground 
CGI device . 

8. Apparatus according to any of the preceding claims 
wherein the control parameters define the three-dimensional 
position of a feature in the background object, and wherein 

15 the interaction means causes* the foreground CGO to be at 
least partially obscured when the monitored position of the 
foreground CGO lies behind the three-dimensional position 
of the feature . 

9. Apparatus according to any of the preceding claims 
2 0 wherein the image data defining a background object 

comprises video data. 

10. A method of interactively generating a display signal, 
the method comprising 

receiving a broadcast signal, the broadcast signal 

2 5 comprising a plurality of datastreams each including image 

data defining a background object, and control parameters; 

generating a foreground computer generated object 
(CGO) ; 

--combining the foreground CGO with the background 

3 0 object image data to generate the display signal; 

monitoring the position of the foreground CGO; and 
adapting the display signal with reference to the 

monitored position of the foreground CGO and the received 

control parameters . 
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applicabilrty; citations and explanations supporting such statement 

1. Statement 

Novelty (N) Yes: Claims 1-12 

No: Claims 

Inventive step (IS) Yes: Claims 1-12 

No: Claims 

Industrial applicability (lA) Yes: Claims 1 -1 2 

No: Claims 

2. Citations and explanations 
see separate sheet 



Form PCT/IPEA/409 (Boxes I- VIII, Sheet 2) (January 1994) 




INTERNATIONAL PRELIMINARY international application No. PCT/GB98/01 798 

EXAMINATION REPORT - SEPARATE SHEET 



Closest prior art is known from "KOENEN R ET AL: •MPEG-4: Context and objectives' 
which describes a system which eases the integration of natural and synthetic audio 
and video material, as well as other type of data. 

The problem to be solved by the present application is to enable the system to process 
also further signal types. 

None of the cited documents describes the solution defined in claims 1 and 9 to 
combine a signal, representing a sequence of video frame signals together with objects 
defining data and control parameters. 

This solution is also not obvious because KOENEN teaches to handle the video signals 
as objects and not in the form of video frame signals. 



Fnm PCT/Seoarate SheeV409 (Sheet 1 ) (EPO-April 1997) 



"MPEG-4: Context and Objections", Koenen et al , Signal 
Processing: Image Communication 9 (1997) pages 295-304 
describes a system which eases the integration of natural 
and synthetic audio and video material, as well as other 
5 data types, such as text overlays and graphics. However, 
the paper admits that this cannot be used with frame-based 
video . 

In accordance with a first aspect of the present 
invention there is provided apparatus for interactively 
10 generating a display signal, the apparatus comprising 

a receiver for receiving a broadcast signal, the 
broadcast signal comprising at least one datastream 
including a sequence of video frames, data defining a 
background object corresponding to each video frame, and 

15 control parameters; 

and a processing system for generating a foreground 
computer generated object (CGO) , for monitoring the 
position of the foreground CGO with respect to the 
background object, and for combining the foreground CGO 

20 with the background object in accordance with the control 
parameters and with the video frame to generate the display 
signal . 

In accordance with a second aspect of the present 
invention there is provided a method of interactively 
25 generating a display signal, the method comprising 

receiving a broadcast signal, the broadcast signal 
comprising at least one datastream including a sequence of 
video frames, data defining a background object 
corresponding to each video frame, and control parameters; 
30 generating a foreground computer generated object 

(CGO) ; 

monitoring the position of the foreground CGO with 
respect to the background object; and, 

combining the foreground CGO with the background 
35 object in accordance with the control parameters and with 
the video frame to generate the display signal. 



r 
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The present invention provides the capability of 
interaction with the actual broadcast itself as it appears 

on a screen in real time. 
5 Typically the method and apparatus is provided for use 
in association with a TV set to provide levels of 
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20 
CLAIMS 



1. Apparatus for interactively generating a display- 
signal, the apparatus comprising 

a receiver for receiving a broadcast signal, the 
broadcast signal comprising at least one datastrean 
including a sequence of video frames, data defining a 
background object corresponding to each video frame, and 

control parameters; 

and a processing system for generating a foreground 
computer generated object (CGO) , for monitoring the 
position of the foreground CGO with respect to the 
background object, and for combining the foreground CGO 
with the background object in accordance with the control 
parameters and with the video frame to generate the display 
signal . 

2. Apparatus according to claim 1 wherein the control 
parameters define the position (s) of one or more areas of 
interaction in the background object, and wherein the 
processing system modifies the display signal when the 
position of the foreground CGO coincides with the position 
of a selected area of interaction. 

3. Apparatus according to claim 2 wherein the control 
parameters define one or more rules associated with the or 
each area of interaction, and wherein the processing system 
modifies the display signal in accordance with the or each 
rule associated with the selected area of interaction. 

4. Apparatus according to any of the preceding claims 
wherein the processing system modifies the display signal 

30 by modifying the foreground CGO. 

5. Apparatus according to any of the preceding claims 
wherein the broadcast signal comprises a plurality of 
datastreams, the receiver being responsive to an upload 
request signal to select one of the datastreams, and 

35 wherein the apparatus further comprises means for inputting 
upload request signals to the receiver in response to input 
from a user. 
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6. Apparatus according to claim 5 wherein the processing 
system modifies the display signal by inputting an upload 
request signal to the receiver. 

7. Apparatus according to any of the preceding claims 
5 further comprising a user operable controller for 

controlling the foreground CGO generated by the processing 
system. 

8. Apparatus according to any of the preceding claims 
wherein the control parameters define the three-dimensional 

10 position of a feature in the background object, and wherein 
the processing system causes the foreground CGO to be at 
least partially obscured when the monitored position of the 
foreground CGO lies behind the three-dimensional position 
of the feature . 

15 9. A method of interactively generating a display signal, 
the method comprising 

receiving a broadcast signal, the broadcast signal 
comprising at. least one datastream including a sequence of 
video frames, data defining a background object 
2 0 corresponding to each video frame, and control parameters; 

generating a foreground computer generated object 

(CGO) ; 

monitoring the position of, the foreground CGO with 
respect to the background object; and, 
25 combining the foreground CGO with the background 

object in accordance with the control parameters and with 
the video frame to generate the display signal . 

10. A method according to claim 9, wherein the broadcast 
signal comprises a plurality of datastreams , the . method 

30 further comprising selecting one of the datastreams to be 
received . 

11. A method according to claim 10, wherein each 
datastream includes a sequence of video frames each 
representing, alternative views relating to a common 

35 subject. 

12. A method according to claim 10 or claim 11, wherein 
the selecting step occurs when the foreground CGO is 
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located at a predetermined position relative to the 
background object. 
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Where a demand for intemationai preliminary examination has bem\/\B filed, see below. 

How? Either by cancelling one or more entire daima, by adding one or more new daims or by amending the text of 

one or more of the daims as filed. 
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International application Is English, the tetter must be In English; If the language of the international appilcatlon 
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Notes to Fomi PCT/ISA/220 (first sheet) (January 1994) 





NOTES TO FORM PCT/ISA/220 (continued) 



The letter must tndtcate the differences between the claims as filed and the claims as amended, ft must, in 
particular, indkate, in connection with each ctarm appearing in the international application (it being understood 
that identical tn<£cation8 concerning several claims may be grouped) .whether 

(i) the claim is unchanged; 

(S) the claim is cancelled; 

(iii) the claim is new; 

fiv) the claim replaces one or more claims as filed; 

(v) the claim is the result of the division of a claim as filed. 



The following examples lltustrate the manner In which amendments must be expl^ned In the 
accompanying tatter: 

1 . [Where originally there were 48 claims and after amendment of some claims there are 51]: 
*Clajma 1 to 29, 31 , 32, 34, 35, 37 to 48 replaced by amerxled claims bearing the same numbers; 
daims 30, 33 and 36 unchanged; new claims 49 to 51 added.* 

2. [Where originally there were 1 5 claims and after amendment of all claims there are 11]: 
'Claims 1 to 1 5 replaced by amended daims 1 to 1 1 J 

3. [Where originally there were 1 4 claims and the amarKlments consist m cancelling some claims and in adcfing 
rtew daims]: 

"Claims 1 to 6 and 14 unchanged; claims 7 to 1 3 canoeDed; new claims 15, 16 and 17 added.* or 
'Claims 7 to 13 cancelled; new claims 1 5, 16 and 17 added; all other claims unchanged.* 

4. [Where various kinds of amendments are made): 

'Claims 1-10 unchanged; daims 11 to 1 3, 1 8 and 1 9 cancelled; claims 1 4, 1 5 and 1 6 replaced by amended 
daim 14; claim 17 subdivided into amerxied daims 15, 16 and 17; new daims 20 and 21 added.' 



"Staiement under article 19(1)" (Rule 46.4) 

The amendments may be accompanied by a statement explaining the amendmenta and tnc£catir>g any impact 
that such amendments might have on the description and the cfrawings (which cannot b>e amended under 
Article 19(1)). 

The statement will be putilished with the intamational application and the amended daims. 
It must be In the language In which the International apppllcation la to be published. 

It must be brief, not exceeding 500 words if in English or if translated into English. 

It should rK>t be confused with and does not replace the letter indicating the cfifferenoes between the claims 
as filed and as aunended. It must be filed on a separate she^ and must t>e identified as such by a heading, 
preferably by using the words 'Statement under Article 1 9(1).* 

It may not contain any disparaging comments on the international search report or the relevance of citations 
contained in that report. Reference to citations, relevant to a given daim, contairrad in the international search 
report may be made only in connection with an amendment of that claim. 



Consequence If a demand for International prellmlnaiy examination has already been filed 

If, at the time of filing any amendments under Article 1 9, a demand for international preliminary examination 
has already t>een submitted, the applicant must preferataly, at the same time of filing ttie amendments with the 
International Bureau, also file a copy of such amendments with the Intemational Preliminary Examining 
Authority (see Rule 62.2(a), first sentence). 



Consequence with regard to translation of the Intemational application for entry Into the national phase 

The applicant's atterrtion is drawn to the fact that, where upon entry into the national phase, a translation of the 
daims as amended uirder Artide 19 may have to tse furnished to the designated/elected Officea, instead of, or 
in addition to, the translation of the daims as filed. 

For further details on the requirements of each designated/elected Office, see Volume II of the PCT Applicarrt's 
Guide. 
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This International Search Report has been prepared by this International Searching Authority and is transmitted to the applicant 
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This International Search Report consists of a total of 



sheets. 



[xl It is also accompanied by a copy of each priorart document cited in tiiis report. 



1. Certain claims were found unsearchable (see Box I). 

2. Unity of invention Is lacking (see Box II). 



3. I I The international application contains disclosure of a nucleotide and/or amino acid sequence listing and the 
international search was carried out on the basis of the sequence listing 

I I filed with the international application. 

I I furnished by the applicant separately from the international application, 

I I but not accompanied by a statement to the effect that it did rrot irrclude 
matter going beyond ttie disclosure in the international application as filed. 
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5. With regard to the abstract, 



□ 



ttie text is approved as submitted by the applicant 

tfie text has been established, according to Rule 38.2(b). by this Authority as it appears in 
Box III. The applicant may, witiiin one montfi from the date of mailing of this International 
Search Report, submit comments to this Authority. 



6. The figure of tiie drawings to be published with the abstract is: 

Figure No. ? as suggested by the applicant. 

I I because ttie applicant failed to suggest a. figure. 
fxl because ttiis figure better characterizes the invention. 



I I None of the figures. 
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cited to understarvj the principle or theory underlying the 
invention 

"X* document of particular relevance: the claimed invention 
cannot be considered novel or cannot be considered to 
involve an inventive step when the document is taken alone 

"Y" document of particular relevarYce; the claimed invention 

cannot be considered to involve an inventive step when the 
document is combined with one or more other such docu- 
ments, such combinatk)n being obvious to a person skilled 
in the art 

*&* document member of the same patent family 



Date of the actual completion of thelntematior^l search 



2 September 1998 



Date of mailing of the intemationai search report 

10/09/1998 



Name and mailing address of the ISA 

European Patent Office, P. B. 581 8 Patentlaan 2 
NL - 2280 HV Riisv/qk 
Tel. (+31-70) 340-2040, Tx. 31 651 epo nl. 
Fax: (+31-70) 340-3016 



Authorized officer 



Sindic, G 



Form PCT/ISA/210 (second sheet) (Juty 1992) 



page 1 of 3 



iNTOiilATlONAL SEARCH (^£POftt 




lnt^aj|onal Application No 

98/01798 



C^Contlnuatlon) DOCUMENTS CONSIDERED TO BE RELEVANT 



Category * 



Citation of document, with tndicatioa where appropriate, of the relevant passages 



Relevant to datm No. 



Y 
A 



DOENGES P K ET AL: "Audio/video and 
synthetic graphics/audio for mixed media" 
SIGNAL PROCESSING. IMAGE COMMUNICATION, 
vol. 9, no. 4, May 1997, page 433-463 
XP004075338 



see 
1 

see 
1 

see 
2 

see 
1 

see 
2 

see 
3 - 
1 

see 
3 



page 434 

page 434 

page 435 

page 437 

page 438 

page 439 
page 441 



left-hand column, paragraph 

right-hand column, paragraph 
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